User interface device responsive to data tag associated with physical location

ABSTRACT

A system includes a display configured to display a graphical simulation, a computer configured to run a software program designed to display the graphical simulation on the display, and a stylus constructed and arranged to be manipulated by a user of the system in at least two dimensions to affect the graphical simulation. The stylus includes an actuator configured to output a haptic effect to the user upon receiving a control signal from the computer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 12/642,323, filed on Dec. 18, 2009, which claimsthe benefit of priority from U.S. Provisional Patent Application Ser.No. 61/139,506, filed on Dec. 19, 2008, the contents of both of whichare incorporated herein by reference in their entireties.

FIELD

The present disclosure relates generally to systems and methods forproviding a haptic effect associated with a graphical simulation orvirtual tag.

BACKGROUND

Video games and video game systems have become even more popular due tothe marketing toward, and resulting participation from, casual gamers.Popular video game and computer systems have painting-based games inwhich regular palm-type controllers or a computer mouse has been used tomove a displayed virtual paintbrush on a virtual canvas. However, thereis no handheld game controller and accompanying software game whichimmerses the user into a realistic gaming experience of painting ordrawing on real or virtual surfaces.

SUMMARY

Embodiments of the present invention disclose a gaming concept in whicheach user in a single-player or a multi-player game is enabled togenerate visual art, which may be real or virtual markings on a real orvirtual surface using haptic-enhanced controllers. The visual art may beartwork or graffiti work that is created virtually, i.e. on virtualsurfaces, or on actual surfaces. The controllers can emulate theexperience of using a marker, a paint brush, a paint spray-can or thelike. The components of various controllers may be modularized foreasily interchanging components to extend the art or graffiti creationexperience. The real life experience of using ink or paint on a surfaceis simulated by haptic feedback and may be enhanced by audio effects.When a paint spray-can controller is used, the experience of paintconsumption is recreated by various peripheral enhancements includingaudio and haptic sensations communicated to the user.

According to an aspect of the invention, there is provided a system thatincludes a computer configured to run a software program designed todisplay a graphical simulation on a display. The software program isconfigured to allow a user to generate visual art in the graphicalsimulation. The system includes a user graspable controller configuredto be freely moveable in a three dimensional space. The controllerincludes a sensor configured to track positional and orientative valuesof the controller in the three dimensional space and transmit a sensorsignal representative of the positional values to the computer, and anactuator configured to output a haptic effect to the user upon receivinga control signal from a processor coupled to the controller. The usergraspable controller is representative of a writing and paintinginstrument configured to generate the visual art.

According to an aspect of the invention, there is provided a game thatincludes a user graspable controller configured to be freely moveable ina three dimensional space by a user. The controller includes a positionsensor configured to track a three-dimensional position, orientation,and geographical location of the controller. The controller isconfigured to place a data tag at a location where the user designatesvia the controller, the controller configured to send a signal after thecontroller places the data tag. The game includes a host computer remotefrom the controller and configured to run a software program. The hostcomputer is configured to identify the user from the received signal andto store the location of the data tag, wherein information of the storeddata tag and identity of the user is capable of being accessed from thehost computer. Other data about the tag may also be stored, such as itsshape, texture, other aspects of its appearance, timestamp, gesture,etc.

According to an aspect of the invention, there is provided a game thatincludes a plurality of uniquely identifiable controllers configured tobe freely moveable in a three dimensional space by respective users.Each controller includes a position sensor configured to track aphysical position of the controller in the three dimensional space. Eachcontroller is configured to place a data tag on a surface at differentuser-designed locations. Each controller is configured to send acorresponding signal upon the controller placing the data tag. The gameincludes a host computer remote from the controllers and configured torun a software program. The host computer is configured to identify eachuser from each received signal and to store the location of the datatag, wherein information of the stored data tag and identity of theusers is capable of being accessed from the host computer.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more examples ofembodiments and, together with the description of example embodiments,serve to explain the principles and implementations of the embodiments,and in which:

FIG. 1A and FIG. 1B illustrate side views of game controllers in theshape of a marker in accordance with embodiments of the invention;

FIG. 2A and FIG. 2B illustrate side views of game controllers in theshape of a paint brush in accordance with embodiments of the invention;

FIG. 3A and FIG. 3B illustrate side views of game controllers in theshape of a paint spray-can in accordance with embodiments of theinvention;

FIGS. 3C-3E illustrate embodiments of detachable top portions of a paintspray-can game controller shown in FIG. 3B;

FIG. 3F illustrates an enhanced paint spray-can game controlleraccording to an embodiment of the invention;

FIG. 3G illustrates a stencil according to an embodiment of theinvention;

FIG. 4 illustrates various detachable components of game controllersaccording to embodiment of the invention;

FIG. 5 illustrates a hybrid type game controller according to anembodiment of the invention;

FIG. 6 illustrates a block diagram of a game controller in use with agaming system in accordance with an embodiment of the invention; and

FIG. 7 illustrates a block diagram of a viewer in accordance with anembodiment of the invention.

DETAILED DESCRIPTION

Example embodiments are described herein in the context of aninteractive painting game and associated controller. Those of ordinaryskill in the art will realize that the following description isillustrative only and is not intended to be in any way limiting. Otherembodiments will readily suggest themselves to such skilled personshaving the benefit of this disclosure. Reference will now be made indetail to implementations of the example embodiments as illustrated inthe accompanying drawings. The same reference indicators will be usedthroughout the drawings and the following description to refer to thesame or like items.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific decisions must be madein order to achieve the developer's specific goals, such as compliancewith application- and business-related constraints, and that thesespecific goals will vary from one implementation to another and from onedeveloper to another. Moreover, it will be appreciated that such adevelopment effort might be complex and time-consuming, but wouldnevertheless be a routine undertaking of engineering for those ofordinary skill in the art having the benefit of this disclosure.

In accordance with this disclosure, the components, process steps,and/or data structures described herein may be implemented using varioustypes of operating systems, computing platforms, computer programs,and/or general purpose machines. In addition, those of ordinary skill inthe art will recognize that devices of a less general purpose nature,such as hardwired devices, field programmable gate arrays (FPGAs),application specific integrated circuits (ASICs), or the like, may alsobe used without departing from the scope and spirit of the inventiveconcepts disclosed herein. It is understood that the phrase “anembodiment” encompasses more than one embodiment and is thus not limitedto only one embodiment. Where a method comprising a series of processsteps is implemented by a computer or a machine and those process stepscan be stored as a series of instructions readable by the machine, theymay be stored on a tangible medium such as a computer memory device(e.g., ROM (Read Only Memory), PROM (Programmable Read Only Memory),EEPROM (Electrically Erasable Programmable Read Only Memory), FLASHMemory, Jump Drive, and the like), magnetic storage medium (e.g., tape,magnetic disk drive, and the like), optical storage medium (e.g.,CD-ROM, DVD-ROM, paper card, paper tape and the like) and other types ofprogram memory.

FIGS. 1A-B, 2A-B, 3A-B and 3F illustrate various haptically enabledwriting/painting instrument controllers for use with a game inaccordance with embodiments of the invention. FIGS. 1A and 1B representtwo embodiments 100 and 150 of marker-type controllers for use with aninteractive game. FIGS. 2A and 2B illustrate two embodiments 200 and 250of paintbrush-type controllers for use with an interactive game. FIGS.3A, 3B, and 3F illustrate different embodiments of paint spray-can-typecontrollers 300, 350 and 385 for use with an interactive game. It shouldbe noted that the controllers shown in the Figures are only examples andany other design is contemplated. In an embodiment, the controller maybe a mobile phone, smart phone, mobile computer, digital camera,personal gaming system (e.g. PSP, Nintendo DS), computer mouse, lightpen, stylus, and the like.

In particular, the marker-type controller 100 shown in FIG. 1A includesa body 102 shaped like a marker having a tip 101. The body 102 of thecontroller has two parts: a tip part 102A, and a main body part 102B.Parts 102A and 102B may be integral or monolithic, as shown inembodiment 100 in FIG. 1A, or comprising detachable modules, as shown inembodiment 150 in FIG. 1B. In the detachable tip embodiment of thecontroller, a suitable coupler may be used to removeably attach the tippart 102A of the body to the main body part 102B. For example, inembodiment 150, pins 110A-B protrude from the tip part 102, that can besnugly coupled to the corresponding complementary slots 112A-B in themain body part 102B. The slots 112A-B may be in the tip part 102A, andthe pins 110A-B may be in the main body part 102B in certainembodiments. The number and position of pins and slots may vary. Othertypes of mechanical and/or magnetic coupling mechanisms may be usedinstead of the pin/slot configuration. However the coupler shouldprovide a conduit for wired (shown in thin lines) and wirelessconnections between the functional blocks (shown with thick outlines)housed within the body 102. Controller 100 also includes one or moresensors 104A-B, one or more actuators 106A-B, a local processor 108coupled to the one or more sensors 104A-B as well as the one or moreactuators 106A-B and an input/output port 110 which may be wired, asshown, or wireless using any appropriate known technology to allow thecontroller 100 to communicate with another device, such as a hostcomputer (e.g. video game console, server, desktop computer) not shownin FIG. 1A. The controller 100 may also include a speaker 140 that iscoupled to the local processor 108 and is configured to project soundsrelated to the use of the controller 100. The controller 100 may beconfigured to house an ink cartridge within, whereby the cartridgesupplies ink to the tip 101 for application onto a surface. In anotherembodiment, the controller 100 is not configured to actually apply inkor paint to a surface and just only applies “virtual ink” or “virtualpaint.”

In an embodiment, the marker controller 100 includes a tip sensor 104Aas well as a body sensor 104B, whereby the tip sensor 104A monitors whenthe tip 101 comes into contact with a real surface and sends a tipsensor signal to the local processor 108 coupled to a local memory 107.The local processor 108 may then indicate to a host computer (e.g., hostcomputer 690 shown in FIG. 6) that the user is ready to apply ink ontothe surface based on the host computer commands or user commands. Itshould be noted that instead of a tip sensor 104A, the controller 100may include a button (not shown) that sends a similar sort of indicationto the host computer that the user is ready to apply ink onto thesurface.

A body sensor 104B may also be incorporated in the controller 100 asshown in FIG. 1A, whereby the body sensor 104B monitors and tracksmovement of the controller in a two or three dimensional space. Such abody sensor may include a two or three dimensional capableaccelerometer, gyroscope or other appropriate type of mechanism. Thebody sensor 104B alternatively uses light, optics, infrared, RFID orother like technology to communicate the position, velocity and/oracceleration values of the controller 100 to the host computer. Inanother embodiment, the controller is used in conjunction with a touchscreen or touch pad. It is contemplated in this embodiment, that the tip101 is in contact with or near the touch surface of the touch screen ortouch pad, whereby movement of the tip 101 is preferably tracked alongthe touch screen or touch pad. In response, the host computerdynamically updates the graphical environment run by the software.

In addition, the controller 100 may include a tip actuator 106A which isconfigured to output a haptic effect upon receiving a control signalfrom the processor 108. The haptic effect may be a pop, click, vibrationor other haptic effect, whereby the actuator 106A outputs an appropriatehaptic effect that provides the user with a realistic feeling orindication of the marker 100 touching a surface.

Additionally or alternatively, a body actuator 106B preferablypositioned within the body 102 of the controller 100 outputs a hapticeffect upon receiving a corresponding control signal from the processor,whereby the body actuator 106B provides the user with a realisticfeeling of the marker being used. Such haptic effects include, but arenot limited to, a vibration, jolt, damping effect, etc. The vibrationoutput by either or both actuators 106A, 106B may depend on the amountof force applied onto the marker by the user, whereby the tip sensor104A senses the amount of force (or distance that the tip 101 moves).For example, the vibration output by either or both actuators 106A, 106Bmay increase as the user presses the marker 100 harder onto the surface,whereas the vibration output by either or both actuators 106A, 106B maydecrease as the user relieves the force applied to the surface. Itshould be noted that the software is programmed to accordingly send thecontrol signals to the tip and/or body actuators based on game play.

With regard to the brush controller 200 embodiment in FIG. 2A, thecontroller is shown to include a body 202 having a bristled tip 201. Thebody of the controller has two parts: a main body part 202B and a tippart 202A. Parts 202A and 202B may be integral or monolithic (as shownin embodiment 200 in FIG. 2A), or comprising detachable modules, asshown in embodiment 250 in FIG. 2B. Complementary fastening mechanisms210A-B and 212A-B are used to couple the main body part 202B with thetip part 202A.

Controller 200 includes one or more sensors 204A-B, one or moreactuators 206A-B, a local processor 208 coupled to the one or moresensors 204A-B as well as the one or more actuators 206A-B and aninput/output port 210 which may be wired, as shown, or wireless usingany appropriate known technology to allow the controller 200 tocommunicate with one or more host computers. The controller 200 may beconfigured to house an ink or paint cartridge within, whereby thecartridge supplies ink or paint to the brush 201 for application to asurface based on the host computer commands or user commands. In anotherembodiment, the controller 200 is not configured to actually apply inkor paint to a surface and just only applies “virtual ink” or “virtualpaint” via the brush 201. It is contemplated that the controller 200 maybe configured to allow the user to dip the bristled tip 201 into paint,whereby the bristled tip 201 acts as a real paintbrush in being capableof applying paint onto a surface. Although not illustrated in theFigures, the controller 200 may also include a speaker, like the speaker140 of the controllers 100, 150 illustrated in FIGS. 1A and 1B, that isconfigured to emit sounds associated with the use of the controller 200.

In an embodiment, the marker controller 200 includes a tip sensor 204Aas well as a body sensor 204B, whereby the tip sensor 204A monitors whenthe bristle tip 201 comes into contact with a real or virtual canvas andthereby sends a tip sensor signal to the local processor 208 (coupled toa local memory 207) and/or host computer (not shown in FIG. 2A-2B).Preferably, the local processor 208 indicates to the host computer thatthe user is engaged in applying paint onto a surface. It should be notedthat instead of a tip sensor 204A, the controller 200 may include abutton (not shown) that sends a similar indication to the host computeronce the user presses the button.

A body sensor 204B may also be incorporated in the controller 200 asshown in FIG. 2A, whereby the body sensor 204B monitors and tracksmovement of the controller in a two or three dimensional space. Such abody sensor may include a two or three dimensional capableaccelerometer, gyroscope or other appropriate type of mechanism. Thebody sensor 204B alternatively uses light, optics, infrared, RFID orother like technology to communicate the position, velocity and/oracceleration values of the controller 200 to the host computer. Inanother embodiment, the controller is used in conjunction with a touchscreen or touch pad. It is contemplated in this embodiment, that the tip201 is in contact with the touch screen or touch pad, whereby movementof the tip 201 is preferably tracked along the touch screen or touchpad. In response, the host computer dynamically updates the graphicalenvironment run by the software.

In addition, the controller 200 may include a tip actuator 206A which isconfigured to output a haptic effect upon receiving an appropriatecontrol signal from the processor 208. The haptic effect may be a pop,click, vibration, texture or other haptic effect, whereby the actuator206A outputs an appropriate haptic effect that provides the user with arealistic feeling of the brush controller 200 touching a surface.

Additionally or alternatively, a body actuator 206B positioned withinthe body 202B of the controller 200 may output a haptic effect whichprovides the user with a realistic feeling of a brush being used. Suchhaptic effects include, but are not limited to, a vibration, jolt,damping effect, etc. The vibration output by either or both actuators206A, 206B may depend on the amount of force applied onto the brush tipby the user, whereby the tip sensor 204A senses the amount of force (ordistance that the brush tip 201 moves). For example, the vibration ortexture output by either or both actuators 206A, 206B may increase asthe user presses the brush controller 200 harder onto the surface,whereas the vibration output by either or both actuators 206A, 206B maydecrease as the user relieves the force applied to the surface.

With regard to the paint spray-can controller embodiment 300 in FIG. 3A,the controller 300 is shown to include a body 302 having a nozzle 301,one or more sensors 304, one or more actuators 306, a local processor308 coupled to a local memory 307, and also coupled to the one or moresensors 304, as well as the one or more actuators 306 , An input/outputport 310, which may be wired, as shown, or wireless using anyappropriate known technology, is included to allow the controller 300 tocommunicate with one or more host computers (not shown). The controller300 may be configured to house a spray paint cartridge within wherebythe cartridge supplies paint which is ejected out of the nozzle 300 forapplication to a surface. In another embodiment, the controller 300 isnot configured to actually apply ink or paint to a surface and just onlyapplies “virtual paint.”

The body of the controller has two parts: a main body part 302B and atip part 302A. Parts 302A and 302B may be integral or monolithic, asshown in embodiment 300 in FIG. 3A, or detachable, as shown inembodiment 350 in FIG. 3B. Complementary fastening mechanisms 310A-B and312A-B are used as a coupler to couple the main body part 302B with thetip part 302A, as shown in FIG. 3B. The detachable tip part 302A mayhave various alternative configurations. FIGS. 3C-3E illustrate threeexample configurations 302C, 302D, and 302E of the tip part 302A.Configurations 302A, 302C, 302D and 302E have different physicalarrangement of the functional blocks (shown in thick outlines) andinternal wiring or wireless connections (shown in thin lines) leading tothe fastening mechanisms 310A-B. Persons skilled in the art willappreciate that the scope of the invention is not limited by thephysical arrangement of the functional blocks and wiring/wirelessconnections. Tip configuration 302E shows two sets of fasteningmechanisms, 310A-B and 310C-D. Fastening mechanisms 310A-B may be usedwhen a regular paint spray can mode is used, and fastening mechanisms310C-D may be used when a blended marker/spray can mode is used, asshown in FIG. 5.

In an embodiment, the controller 300 includes a nozzle sensor 304A aswell as a body sensor 304B. The nozzle sensor 304A preferably monitorswhen the nozzle 301 is pressed by the user, although it is possible forthe sensor 304A to measure the amount of distance that the nozzle 301 ispressed or released by the user. The nozzle sensor 304A is configured tosend a nozzle sensor signal to the local processor 308 and/or hostcomputer indicating when the nozzle is pressed 301 and/or how far nozzleis pressed by the user. Preferably, the local processor 308 indicates tothe host computer (not shown) if and how far along the nozzle 301 ispressed. The software allows the host computer to correspondinglydisplay the virtual sprayed paint on the displayed surface. It ispossible for the software to represent the paint becoming more intenseor less intense based on far down the user pushes the nozzle 301 (e.g.lighter showing of paint when nozzle is lightly pressed). It should benoted that instead of a nozzle sensor 304A, the controller 300 mayinclude another type button (not shown) that sends a similar indicationto the host computer once the user presses the button.

In addition, the controller 300 may include a nozzle actuator 306A whichis configured to output a haptic effect upon receiving a correspondingcontrol signal from the processor 308. The haptic effect may be a pop,click, vibration or other haptic effect, whereby the actuator 306Aoutputs an appropriate haptic effect to the nozzle that provides theuser with a realistic feeling via the user's digit that the can 300 isbeing operated. It is contemplated that the nozzle actuator 306B orseparate device produces an opposing force as the nozzle 301 is presseddown.

Additionally or alternatively, a body actuator 306B preferablypositioned within the body 302 of the controller 300 outputs a hapticeffect which provides the user with a realistic feeling of the can 300being used. The vibration output by either or both actuators 306A, 306Bmay depend on the amount of “virtual spray paint” comes out of thenozzle. For instance, the vibration produced by the nozzle and/or bodyactuators would preferably have a greater magnitude when the nozzle 301is sensed pressed at or near a predetermined threshold distance (e.g.furthest distance from neutral) as opposed to a lighter magnitude whenthe nozzle 301 is lightly pressed.

The basic paint spray-can embodiments of FIGS. 3A and 3B may be enhancedby adding a number of functional components as peripherals or asintegrated/embedded devices within the controller. The added componentsfacilitate in enhanced user experience. For example, FIG. 3F shows anenhanced paint spray-can embodiment 385, which includes a shakablerattle/ball 325, a co-located speaker 340, an air compressor 335, atailored motor 330, and optionally a GPS device 345, and a camera 355.Persons skilled in the art will understand that not all of these devicesmay be included in a single embodiment, and different embodiments mayhave different combinations of the additional components.

In an embodiment, air compressor 335 or other like device expels air outof the nozzle 301 when the user presses down on the nozzle 301. Indischarging the air, the air compressor 335 causes the spray-cancontroller to feel as if it is becoming lighter. Air compressor 335 maybe connected to the speaker 340, so that the air release sound isamplified. Audio is often a key enhancer of user experience. Therefore,in an embodiment, co-located speaker 340 is included. Other features maybe included on the spray-can controller to make the experience of usingthe controller more realistic. For instance, the controller 385 mayinclude a shakable rattle/ball 325 within its body to mimic a realspray-can when it is shaken. The sound of the rattle being shaken can beamplified by the co-located speaker 340. Different viscosity feeling maybe simulated and associated with the rattle shaking as more paint isconsumed, and paint level in the can is getting lowered. Hapticsfeedback are generated by tailored motors 330 coupled to actuator 306Bfor simulating a realistic spray-can use experience. Further, thespray-can controller may include a Peltier or other cooling devicecoupled to the outer surface which makes the outer surface of thespray-can controller feel cooler as more air is discharged.

GPS 345 and camera 355 may enhance the experience in an onlinemulti-player game, as will be discussed with reference to Figure. 7.

FIG. 3G shows a front view of an external fitted cap 322 with a stencil303 that may be used in association with the nozzle 301, when properlyaligned. In FIG. 3F, the side view of the cap 322 and the alignedstencil 303 are shown. Persons skilled in the art will appreciate thatdifferent stencil shapes can be accommodated either by coupling a realcap like cap 322, or by simulating a stencil shape by software. Usersmay be given the flexibility of choosing a preferred stencil shape froma suite of available shapes

It is further conceived that the controllers might be modularized intothree main components, tip part, main body part and core circuit, asshown in the examples in FIG. 4. FIG. 4 shows various tip parts, suchas, paint brush tip part 202A (FIG. 4(C)), and marker tip part 102A(FIG. 4(D)), various main bodies, e.g. pain brush body 202B (FIG. 4(A))and marker body (FIG. 4(B)), and various core circuits, e.g., paintbrush core circuit 400A (FIG. 4 (E)), and marker core circuit 400B (FIG.4(F)), This type of modularization of the controllers would allow formixing modifying components for new hardware configurations and userexperiences without having to switch to other controllers. This modularcomponent system would also allow for such things as housing smallercontroller bodies within larger bodies; for instance having a markerbody housed inside a spray can housing such that the user could removethe marker tip for quick interactions in concert with the spray canbody. An embodiment of a marker body inserted into a spray can body isshown in the spray can embodiment 500 in FIG. 5. Note that the internalwiring and fastening mechanism configuration of tip portion of the body302E supports an operation in conjunction with a marker controller body102B and a marker controller core circuit 400B

In any or all of the above described controllers, the controller mayinclude one or more directional pads thereon which are configured tooperate various features during game play. For instance, the directionalpads may be used to manipulate the user's simulated character and/or thedisplayed brush or spray-can in the graphical environment.

FIG. 6 illustrates a block diagram of a controller 600 in use with ahost computer 690 running a software program. For sake of brevity of thedescription hereinafter, the term “controller” will be discussedgenerally as controller 600, although it should be noted that any of theabove described controllers, i.e., the marker controller, the paintbrush controller, the spray can controller or a combination thereof, maybe used as well.

With regard to the one or more actuators of the controller, the localprocessor 608 preferably provides haptic effect signals to the actuators606A-B based on high level supervisory or streaming commands from thehost computer 690. In the streaming embodiment, the voltage magnitudesand durations are streamed to the controller 600 where information isprovided by the host computer 690 to the actuators 606A-B. In operation,the host computer 690 may provide high level commands to the localprocessor 608 such as the type of haptic effect to be output (e.g.vibration, jolt, detent, pop, etc.) by one or more selected actuators,whereby the local processor 608 instructs the actuators 606A-B as toparticular characteristics of the haptic effect which is to be output(e.g. magnitude, frequency, duration, etc.) The haptic effects therebyprovide the user with the feeling that the controller 600 feels similarto an actual writing/painting device.

In an embodiment, the host computer 690 may be a dedicated video gamingsystem, such as a Playstation, Xbox, or Wii. In an embodiment, the hostcomputer 690 may be or be coupled to one or more servers incommunication with the controller 600, whereby at least a portion of thesoftware program is run on one or more of the servers and/or the hostcomputer. In an embodiment, the software is configured to allow onlinegaming or Massively Multiplayer Online Gaming (MMOG), whereby thesoftware may be operated on several remote and/or local serverssimultaneously.

In an embodiment, the software may be used to allow the user to producea virtual painting or drawing by tracking the user's movements of thecontroller 600 in two or three dimensions using the body and/or tipsensors 604A-B. For instance, the software may display a virtual orgraphical representation of the user's marker, paintbrush or spray-canin front of a surface (i.e. first person view), whereby the softwarecorrespondingly moves a graphical representation of the user's marker,paintbrush or spray-can in response to the tracked movements of thecontroller 600. The software preferably allows the user to selectdifferent colors and types of writing/painting instruments (e.g. pen,charcoal, paint brush, etc.) using the controller 600 and causes thegraphical hand and associated writing/painting tool to mimic the user'sreal life movements, including fluid natural movements of hand in arcsand arbitrary trajectories at various speeds. In another instance, thesoftware may display a virtual or graphical representation of an avataror a game character (i.e. third person view) which the user controls viathe controller 600, such as using a directional pad. In this example,the software may also allow the user to toggle between controlling theuser's character (i.e. third person view) as well as thewriting/painting tool (i.e. first person view) with sufficient accuracy.Haptics may further enhance the experience. For instance, the softwaremay compare the user's gesture(s) via the controller 600 with a one ormore predetermined stored gesture(s). In an embodiment, the controller600 be programmed to output a haptic effect (e.g. low frequency andmagnitude vibration) as long as the user's gestures follow the storedgestures. Alternatively, the actuator(s) in the controller 600 may beprogrammed to output a haptic effect (e.g. jolts or low magnitudevibration) when the user's gestures deviate from the stored gestures towhich they are compared to. Such use of haptics could be used toeffectively train the user on certain painting techniques (e.g. applyingbroad strokes vs. short strokes) or even certain styles (e.g.impressionist vs. abstract).

In an embodiment, the software is designed to be a video game whichallows the user to navigate through and explore a three dimensional openarea city environment (e.g. Grand Theft Auto, Second Life) in which theuser utilizes the controller 600 to navigate around the environment anddraw or paint on virtual structures (e.g. water towers, train cars,government buildings, etc.) displayed in the environment. In thisembodiment of the game, an objective may be to “tag” designatedstructures in the city, based on a mission. A “tag” may be the user'sname, logo, symbol or design (hereinafter generally referred to as“design”) which the user produces or designates. In an embodiment, thesoftware allows the user to manually draw the design using thecontroller 600. The user is alternatively able to select a storedpre-drawn design, whereby the software automatically retrieves thestored design and tags it on the wall or structure whenever the userdesires. Additionally or alternatively, the user can redraw the tag ordraw a different design whenever it is desired. The game can be designedto award points based on the number of places where the user “tags” awall or structure, whereby players gain fame (or infamy) based on thenumber of accumulated points.

The software may have a feature which allows online or offlinemultiplayer action in which other players can place their tag over anexisting tag to gain more points and/or reduce the number of pointsawarded to the player who was “over-tagged”. In an embodiment, thesoftware preferably keeps track of the locations in the virtualenvironment where each of the users have tagged structures and puts atime stamp on each tag. The software thereby is able to award (or takeaway) points based on the stored locations, type of design, and/or thefrequency of tags (or over-tags) within a period of time. Although thesoftware may be designed such that the game is sandbox style, althoughthe game may additionally or alternatively be designed to be missionbased.

As described above, the software provides a virtual open environment inwhich the user is able to use the controller 600 to place virtual tagson virtual buildings. In another embodiment, the user is able to use thecontroller 600 in the real world to tag actual surfaces and structuresusing “invisible ink” or “invisible paint”. In this embodiment, thecontroller 600 may include a global positioning system (GPS) modulecoupled to the transceiver which provides the user's location to one ormore servers and/or host computers 690 when a structure is tagged. Thisuploaded information would preferably include the user's identity, alocation of the tag, a time stamp of when the tag occurred, and a photo,description or other information of the tagged design itself.

In an embodiment, the controller 600 allows the user the freedom topaint or draw whatever is desired using the invisible ink. Inparticular, the software and controller 600 allow the user to takeadvantage of the user's artistic talents in replicating the user'spainting or marking gestures to produce a finished art piece based onthose gestures. In particular, the tip and/or body sensors 604A, 604B ofthe controller 600 track the motion of the controller 600 as the user isdrawing or painting on an actual surface, whereby the processor storesthe tracked information and processes the information to produce areplica of what the user had just drawn based on the positional valuesof the controller 600. In an embodiment, the information can be viewedimmediately on the controller (or other device, such as the displayscreen 642 of the host computer 690) to show what the user's finishedart work looks like by itself or virtually overlaid on the surface. Aspeaker 640A coupled to the host computer 690, or a speaker 640Bco-located with the controller 600 may enhance the user experience byintegrating sound effects.

It is also contemplated that the tag data is uploaded to one or morecentral servers and/or host computers in which the user can access,view, and compare the location and frequency of tags with other users inthe system. The software preferably tracks tag data for every registereduser, in which the users may access the software from their hostcomputer or mobile device to see their tag history as well as otheruser's tag histories in the location. In particular, the software canpair up with existing mobile application software (e.g. Google Maps) inwhich the user can see on the mobile application the areas which havebeen tagged by one or more registered users. The software preferablyallows the user to select on a particular “tag point” to see theidentity of the user, the time which the “tag point” was tagged as wellas be able to retrieve a photo or representation of the art work. Thesoftware would preferably allow the user to see how many points ortagging locations the user has accumulated (or lost due toover-tagging). The software is also capable of handling socialnetworking applications.

In an embodiment, the software will allow users to share and comparetheir art or creations in a social-networked environment. Users will beable to find artists creating similar pieces, and the software will beable to make recommendations to view certain pieces by other artistsbased on interest levels, user profiles, and previously created art.

In an embodiment, the software will allow users to purchase virtualgoods, in form of different colors, materials, brushes and otherpainting materials, as well as street artist materials, in form of spraycans, spray caps, etc.

In an embodiment, the software is configured to work with a viewer orcamera device which allows an individual to actually visit a physicallocation where an item or structure has been tagged and view the artwork through the viewer. FIG. 7 illustrates a block diagram of theviewer in accordance with an embodiment. As shown in FIG. 7, the viewer700 preferably includes a processor 708, a GPS transceiver 745 coupledto the processor 708, a display 742 coupled to the processor 708, anoptional speaker 740, and an optional actuator 706 coupled to theprocessor 708. Preferably, the viewer 700 includes a memory or otherstorage device 707 which stores at least the received information (i.e.tag data). In an embodiment, the viewer is a dedicated device whereas itis contemplated that the same features of the software may be loadedonto a device already having the hardware components shown in FIG. 7(i.e. a mobile phone).

The GPS transceiver preferably communicates the location of the viewerwith the host computer 790 or servers 795 via antenna 750 sendingsignals to a transponder 780, whereby either the host computer 790 orthe viewer 700 compares the location of the viewer with recorded taglocations. Upon the viewer 700 being instructed that it is at a locationof a tagged item or structure, the viewer is notified (visually,audibly, and/or by haptic effect). In an embodiment, the user is able topush a button or use a voice command to view the actual tagged art viathe display 742 whereas the tagged art would normally be invisible tothe naked eye.

The above described controllers and gaming configurations can becombined into a comprehensive game referred to as an “Art Hero” or a“Graffiti Hero”, akin to the popular “Guitar Hero” game. Art Hero may bea gentler version of the Graffiti Hero game. Art Hero games may be moresuited towards entertainment, training, creative pursuit etc., whileGraffiti Hero may be more suited towards high-excitement multi-playergaming. In either Art Hero or Graffiti Hero, the user will preferablyhave a haptic-enhanced controller (paint brush, marker, spray can orother types of writing/painting instruments), and while actually orvirtually painting on a game surface external to the controller, theuser will have the realistic sensation of paint being used on a surface,the spray can being progressively lighter, the colors getting richerwhen a spray nozzle is being pressed harder, etc. In this type of games,haptic feedback can be used to create desired texture or signaturepatterns for the artist. The writing/painting concepts can be integratedto real-life adrenalin-driven experiences, such as the experience ofplaying Grand Theft Auto. For example, Graffiti Hero game may allow auser to work more efficiently under pressure when a “virtual” policeraid is imminent. Various Graffiti artists may also compete against eachother for fame and bragging rights by covering as much as buildingspossible with their respective graffiti, or over-writing on acompetitor's graffiti. In super-enhanced ultimate editions of the games,the users may be allowed to inhabit a virtual world similar to “SecondLife”.

While embodiments and applications have been shown and described, itwould be apparent to those skilled in the art having the benefit of thisdisclosure that many more modifications than mentioned above arepossible without departing from the inventive concepts disclosed herein.The invention, therefore, is not to be restricted except in the spiritof the appended claims.

1-19. (canceled)
 20. A user interface device, comprising: a globalpositioning system (GPS) receiver; a haptic actuator configured togenerate a haptic effect at the user interface device; a display device;and a processor in communication with the GPS receiver, the hapticactuator, and the display device, and configured: to determine, with theGPS receiver, a current location of the user interface device, todetermine whether the current location of the user interface device issubstantially the same as a physical location associated with a data tagstored on a remote host computer, wherein the data tag comprises avisual design or a photo, to cause the haptic actuator to generate thehaptic effect in response to a determination that the current locationof the user interface device is substantially the same as the physicallocation associated with the data tag, and to cause the display deviceto display the visual design or the photo of the data tag in response tothe determination that the current location of the user interface deviceis substantially the same as the physical location associated with thedata tag.
 21. The user interface device of claim 20, further comprising:a camera configured to capture an image of a physical structure or aphysical surface, wherein the processor is configured to cause thedisplay device to display the visual design or the photo by overlayingthe visual design or the photo on the image of the physical structure orthe physical surface.
 22. The user interface device of claim 20, whereinthe user interface device is configured to communicate the currentlocation of the user interface device to the remote host computer, andis configured to receive from the remote host computer an indicationthat the current location of the user interface device is substantiallythe same as the physical location associated with the data tag stored onthe remote host computer, and wherein the determination of whether thecurrent location of the user interface device is substantially the sameas the physical location associated with the data tag is based on theindication received from the remote host computer.
 23. The userinterface device of claim 22, wherein the physical location is one of aplurality of physical locations associated with a plurality ofrespective data tags stored on the remote host computer.
 24. The userinterface device of claim 20, wherein the processor is furtherconfigured to receive from the remote host computer informationdescribing the physical location associated with the data tag stored onthe remote host computer, and wherein the determination of whether thecurrent location of the user interface device is substantially the sameas the physical location associated with the data tag is made bycomparing, at the user interface device, the current location of theuser interface device with the physical location associated with thedata tag.
 25. The user interface device of claim 20, wherein the userinterface device is configured to display the visual design of the datatag, and wherein the visual design comprises one of a logo, a symbol,and a drawing.
 26. The user interface device of claim 20, wherein theprocessor is further configured to place an additional data tag at thecurrent location by generating an additional visual design based on userinput received at the user interface device, and to communicate theadditional visual design and the current location of the user interfacedevice to the remote host computer.
 27. The user interface device ofclaim 26, further comprising: a motion sensor configured to track motionof the user interface device in three dimensional space, wherein theuser input is received via the motion sensor.
 28. The user interfacedevice of claim 26, wherein the processor is configured to furthercommunicate a user identifier to the remote host computer as part ofplacing the additional data tag.
 29. The user interface device of claim20, wherein the user interface device is a mobile phone configured tocommunicate wirelessly with the remote host computer.
 30. A methodperformed by a user interface device having a global positioning system(GPS) receiver, a haptic actuator, a display device, and a processor,the method comprising: determining, by the processor and based on asignal from the GPS receiver, a current location of the user interfacedevice; determining, by the processor, whether the current location ofthe user interface device is substantially the same as a physicallocation associated with a data tag stored on a remote host computer,wherein the data tag comprises a visual design or a photo; causing, bythe processor, the haptic actuator to generate a haptic effect at theuser interface device in response to a determination that the currentlocation of the user interface device is substantially the same as thephysical location associated with the data tag; and causing, by theprocessor, the display device to display the visual design or the photoof the data tag in response to the determination that the currentlocation of the user interface device is substantially the same as thephysical location associated with the data tag.
 31. The method of claim30, wherein the user interface device further comprises a cameraconfigured to capture an image of a physical structure or a physicalsurface, and wherein the step of causing the display device to displaythe visual design or the photo comprises causing the display device tooverlay the visual design or the photo on the image of the physicalstructure or the physical surface.
 32. The method of claim 30, furthercomprising: communicating the current location of the user interfacedevice to the remote host computer; and receiving from the remote hostcomputer an indication that the current location of the user interfacedevice is substantially the same as the physical location associatedwith the data tag, wherein the step of determining whether the currentlocation of the user interface is substantially the same as the physicallocation associated with the data tag is based on the indication fromthe remote host computer.
 33. The method of claim 32, wherein thephysical location is one of a plurality of physical locations associatedwith a plurality of respective data tags stored on the remote hostcomputer.
 34. The method of claim 30, further comprising: receiving fromthe remote host computer information describing the physical locationassociated with the data tag stored on the remote host computer, andwherein the step of determining whether the current location of the userinterface device is substantially the same as the physical locationassociated with the data tag comprises the processor comparing thecurrent location of the user interface device with the physical locationof the data tag.
 35. The method of claim 30, wherein the visual designof the data tag is displayed, and comprises one of a logo, a symbol, anda drawing.
 36. The method of claim 30, further comprising: placing anadditional data tag at the current location by generating an additionalvisual design based on user input received at the user interface device;and communicating the additional visual design and the current locationof the user interface device to the remote host computer.
 37. The methodof claim 36, wherein the user interface device further comprises amotion sensor configured to track motion of the user interface device inthree dimensional space, and wherein the user input is received via themotion sensor.
 38. The method of claim 36, further comprising:communicating a user identifier to the remote host computer as part ofplacing the additional data tag.
 39. The method of claim 30, wherein theuser interface device is a mobile phone configured to communicatewirelessly with the remote host computer.